Papers with Reward Models
Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling (2025.emnlp-industry)
Copied to clipboard
Xiaoyu Liu, Di Liang, Hongyu Shan, Peiyang Liu, Yonghao Liu, Muling Wu, Yuntao Li, Xianjie Wu, Li Miao, Jiangrong Shen, Minlong Peng
| Challenge: | Generative RMs (GRMs) lack contextual and background information during inference, leading to incomplete evaluations. |
| Approach: | They propose a modular and interpretable framework that integrates side-branch models as auxiliary feature generators. |
| Outcome: | The proposed framework outperforms scalar and saline reward models in robustness and alignment with human preferences. |
EQA-RM: A Generative Embodied Reward Model with Test-time Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing generic Reward Models are ill-equipped for dynamic and interactive domains. |
| Approach: | They propose a novel generative multimodal reward model specifically architected for EQA that provides interpretable, structured reward feedback. |
| Outcome: | The proposed model outperforms proprietary benchmarks, including Gemini-2.5-Flash, GPT-4o, Claude-3.5-Haiku, and open-sourced state-of-the-art models such as RoVRM and VisualPRM. |
RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Andrei Catalin Coman, Ionut Teodor Sorodoc, Leonardo F. R. Ribeiro, Bill Byrne, James Henderson, Adrià de Gispert
| Challenge: | Existing Reward Models (RMs) struggle in Retrieval Augmented Generation settings. |
| Approach: | They propose a method that repurposes question-answering datasets into preference pairs that prioritise groundedness over stylistic features. |
| Outcome: | The proposed method surpasses existing RMs trained on larger general corpora with an absolute improvement of +15.5%. |
Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evolve into agentic systems capable of autonomous tool invocation and complex reasoning. |
| Approach: | They propose a trajectory-level preference benchmark to evaluate judges' ability to distinguish preferred versus distractor agent trajectories in tool-integrated environments. |
| Outcome: | The proposed benchmark evaluates how well judges distinguish preferred versus distractor agent trajectories in complex tool-using scenarios. |
RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026.acl-long)
Copied to clipboard
Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, Chen Ma
| Challenge: | Existing benchmarks lack discriminative complexity and ground-truth rubric annotations required for rigorous evaluation. |
| Approach: | They propose a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-based evaluation. |
| Outcome: | The proposed benchmarks show that they support diverse domains, exhibit discriminative ability, provide high-quality annotations, and include human-authored rubrics. |
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026.acl-long)
Copied to clipboard
Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, null Wangyan, Zhang Yueyang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi
| Challenge: | Existing methods for generating reward models focus on outcome-level supervision, neglecting analytical process quality, which constrains their potential. |
| Approach: | They propose a novel reward model that leverages self-reflection to assess analytical quality and enhance preference modeling. |
| Outcome: | The proposed model improves performance on four benchmarks and significantly mitigates positional bias. |